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CROSS-REFERENCE TO RELATED APPLICATIONS 

This application is a continuation-in-part of U.S. patent application serial number 
10/316,672 entitled "Method Of Hidden Surface Reconstruction For Creating Accurate Three- 
Dimensional Images Converted From Two-Dimensional Images" filed on December 10, 2002, 

10 which is a continuation-in-part of U.S. patent application serial number 10/147,380 entitled 
"Method For Conforming Objects To A Common Depth Perspective For Converting Two- 
Dimensional Images Into Three-Dimensional Images" filed on May 15, 2002, which is a 
continuation-in-part of U.S. patent application serial number 10/029,625 entitled "Method And 
System For Creating Realistic Smooth Three-Dimensional Depth Contours From Two- 

15 Dimensional Images" filed on December 19, 2001, now U.S. Patent No. 6,515,659, which is a 
continuation-in-part of U.S. patent application serial number 09/819,420 entitled "Image 
Processing System and Method for Converting Two-Dimensional Images Into Three- 
Dimensional Images" filed on March 26, 2001, which is a continuation-in-part of U.S. patent 
application serial number 09/085,746 entitled "System and Method for Converting Two- 

20 Dimensional Images into Three-Dimensional Images" filed on May 27, 1998, now U.S. 
Patent No. 6,208,348, all of which are incorporated herein by reference in their entirety. 

BACKGROUND OF THE INVENTION 

In the process of converting a two-dimensional (2D) image into a three-dimensional 
25 (3D) image, at least two perspective angle images are needed independent of whatever 
conversion or rendering process is used. In one example of a process for converting two- 
dimensional images into three-dimensional images, the original image is established as the 
left view, or left perspective angle image, providing one view of a three-dimensional pair of 
images. In this example, the corresponding right perspective angle image is an image that is 
30 processed from the original image to effectively recreate what the right perspective view 
would look like with the original image serving as the left perspective frame. 
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In the process of creating a 3D perspective image out of a 2D image, as in the above 
example, objects or portions of objects within the image are repositioned along the 
horizontal, or X axis. By way of example, an object within an image can be "defined" by 
drawing around or outlining an area of pixels within the image. Once such an object has 
5 been defined, appropriate depth can be "assigned" to that object in the resulting 3D image by 
horizontally shifting the object in the alternate perspective view. To this end, depth 
placement algorithms or the like can be assigned to objects for the purpose of placing the 
objects at their appropriate depth locations. 

In a motion picture, objects typically move (i.e., change position within the image, 
10 from frame to frame). An unfortunate consequence of some processes for converting two- 
dimensional images into three-dimensional images is that noticeable visual artifacts appear 
(e.g., as flickering or shuttering pixels) at object edges as objects move from frame to frame. 

In view of the foregoing, it would be desirable to be able to convert two-dimensional 
motion pictures into three-dimensional motion pictures in such a manner that motion 
1 5 artifacts are significantly minimized or eliminated. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Detailed description of embodiments of the invention will be made with reference to 
the accompanying drawings: 
20 FIG. 1 A illustrates an example image with a magnified view of the spacing differences 

between an actual object edge and an object pathline; 

FIG. IB illustrates a sequence of frames representing motion of a person and how 
object to pathline spacing relationships should remain constant as the object moves from frame 
to frame; 

25 FIG. 1C is a graph providing a visual representation of object to pathline spacing 

relationship errors and corresponding levels of perceptible motion artifacts; 

FIG. 2A is an illustration of an arbitrary object showing the object edge and its 
pathline; 

FIG. 2B is an illustration of the object of FIG. 2A showing change of the object shape 
30 at a different point in time; 
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FIG. 3A illustrates an image with two objects, a person and a car, transitioning into the 
image frame; 

FIG. 3B illustrates the same image as FIG. 3A with the dome, wall, and street clock 
objects drawn as Virtual Objects; 
5 FIG. 4A illustrates the same image as FIG. 3A except with the person and car objects 

drawn as Virtual Objects; 

FIG. 4B illustrates an image subsequent to the image of FIG. 3 A with both objects, the 
person and the car, fully in the image frame; 

FIG. 5A illustrates the same image as FIG. 3A except with the person and car objects 
1 0 drawn as Virtual Objects with the car object including a Sub-Object; 

FIG. 5B illustrates the same image as FIG. 4B except with the back lid of the car being 
defined as a Virtual Object; 

FIGs. 6A-6F illustrate how points and/or paths can be applied to objects for creating 
crests, troughs or the like during the process of depth contouring objects; 
15 FIGs. 7A and 7B show examples of inconsistent topology and consistent (or 

preserved) topology, respectively; 

FIGs. 8A and 8B show examples of inconsistent surface shaping and consistent 
surface shaping, respectively; 

FIG. 9A illustrates an interpolative mapping, M, for mapping from one "key frame" 
20 to the next according to an example embodiment of the present invention; 

FIG. 10 illustrates clamping data to the boundaries of objects according to an 
example embodiment of the present invention; and 

FIG. 1 1 illustrates an exemplary system and workstation for implementing the image 
processing techniques of the present invention. 

25 

DETAILED DESCRIPTION 

The following is a detailed description for carrying out the invention. This 
description is not to be taken in a limiting sense, but is made merely for the purpose of 
illustrating the general principles of the invention. 
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The present invention addresses the problem of image artifacts that can result from 
2D to 3D motion picture image conversion processes. It has been observed that noticeable 
image artifacts sometimes occur when object edges are not defined consistently from frame 
to frame during such conversion processes. Accordingly, various methods of the present 
5 invention pertain to providing edges that define objects for converting two-dimensional 
images into three-dimensional images in a manner that is consistent from frame to frame (i.e., 
as the objects move over time). 

In an embodiment of the present invention, a method for converting two-dimensional 
images into three-dimensional images includes: tracking an object in an image, the object 
1 0 having an object edge and an object outline thereabout, from frame to frame over a sequence of 
images; and creating object outlines in subsequent frames for converting two-dimensional 
images into three-dimensional images maintaining an object outline to object edge distance 
spacing relationship as the object moves or changes from frame to frame. 

1 5 Motion Edge Artifacts and Correction: 

As discussed above, visual artifacts in the form of inconsistencies at the edges of 
objects can result from conventional processes of converting 2D motion pictures into 3D 
motion pictures. These inconsistencies can become very noticeable with image object motion 
from frame to frame. It has been observed that such visual artifacts are caused by 

20 inconsistencies in the spacing between object outlines (or pathlines) and the actual edges of 
objects. (For purposes of this description, the terms "outline" and "pathline" are one and the 
same and interchangeable.) Various embodiments of the present invention exploit this 
observation and address the problem of noticeable (objectionable) artifacts resulting from 2D 
to 3D conversion processes by applying image data tracking techniques to maintain actual 

25 object edge to object pathline spacing relationships. 

Thus, in an embodiment of the present invention, a method for converting two- 
dimensional images into three-dimensional images includes: defining an area of pixels around 
an object within an image frame of an image sequence, resulting in an outline of the object 
spaced a particular distance from edges of the object; tracking the object as the object moves 

30 or changes from frame to frame; and re-creating the outline in a different frame of the image 
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sequence maintaining relative distances between the edges of the object and the outline to 
create a different perspective for a three-dimensional image. In another embodiment of the 
present invention, a method for converting two-dimensional images into three-dimensional 
images includes: defining an object within an image frame of an image sequence, resulting in 
5 an outline of the object spaced a particular distance from a perimeter of the object; tracking 
the object as the object moves or changes from frame to frame; and re-creating the outline in 
different frames of the image sequence maintaining relative distances between the perimeter 
of the object and the outline to create an alternate perspective for a three-dimensional image 
sequence. 

1 0 Image data tracking according to the present invention can involve both edge detection 

and optical flow techniques. By tracking image data, a system can be implemented that is not 
only capable of maintaining object edge to object pathline spacing relationships as objects 
move over time, but also pertinent object information (e.g., depth placement information) 
pertaining to Virtual Objects, Sub-Objects, and Contour Points and Pathlines, as described in 

1 5 subsequent sections. 

Referring to FIG. 1 A, according to various embodiments of the present invention, the 
creation of accurate, artifact-free 2D to 3D motion picture conversions involves maintaining 
a near constant difference error, n, between the image object boundary and the associated 
object pathlines. Generally, and also as shown in FIG. 1, pathlines that define objects are 

20 slightly outside of the image object boundary in order to capture the image object in its 
entirety along with the smallest amount of background pixels behind the image object to 
allow for Pixel Repeat, if applicable. See, e.g., U.S. patent application serial number 
09/819,420 (describing "Repeat Pixel" feature: missing gap of pixels resulting from 
repositioning image object along the horizontal, or X axis can be filled by repeating pixels 

25 from the edge of the background object that the object is moving away from). 

Referring to FIG. IB, according to various embodiments of the present invention, as 
an object moves across the image over time, changes in the difference between the object 
edge and the defined pathline of the object are minimized with respect to the difference 
value at the initial frame time. If the object edge to object pathline distance spacing 

30 relationship becomes inconsistent as objects move, it has been observed that this can 
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produce noticeable and distracting artifacts in the image. Generally, the greater the variation 
in the spacing between the image object and its pathline over time (from frame to frame), the 
more noticeable the artifacts become. 

Referring to FIG. 1C, the graph illustrates the relationship between object to pathline 
5 error and the perceptibility of artifacts. The progression of object movement, frame by 
frame, is plotted along the X-axis. The level of artifact perceptibility (or noticeability) is 
plotted along the Y-axis. A greater number along the Y-axis represents a greater difference 
in object to pathline distance, which in turn represents a level of visual artifacts. In the 
illustrated example, the defined pathline for the object is varying in its distance from the 

10 edges of the object from frame to frame over time. If, on the other hand, and in the ideal 
scenario, a particular object has its outline defined consistently frame to frame over time 
relative to the actual object edge, there would be zero error on the graph along the bottom of 
the X axis and the object edges would be quiet, in a manner of speaking, not drawing 
attention to the object. The foregoing presumes that Hidden Surface Areas have also been 

15 appropriately corrected. For purposes of this description, "Hidden Surface Areas" refer to 
areas missing edge information. These areas cause a visual effect in an image (i.e., 
stimulating a different sensation) when a viewer focuses attention on them due to the fact 
they are only seen by one eye. The visual effect is caused by perspective differences at the 
edges between foreground and background objects. See, e.g., U.S. patent application serial 

20 number 1 0/3 1 6,672 (describing "Hidden Surface Reconstruction" techniques). 

Sufficiently large variations in the difference value may cause noticeable 
inconsistencies in object image boundaries. The solution according to various embodiments of 
the present invention is to ensure that object pathlines obey a maximum error threshold relative 

to the initial frame of definition. Assume that for a frame f. in a sequence F = {/)}^ 0 

25 boundary of a region of an image object region can be represented as a simply connected 

closed path p? € 9? 2 parameterizable on [0,1] and that the defined simply connected pathline 

p) e 9i 2 of the object is parameterizable on [0, 1]. Then for frame f t , an error can be defined 
as: 



e i = 



o i 
Pi ~Pi 
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Now for the frame, f Q , of the initial definition of the object, the following can be enforced: 



for all frames in the sequence containing the object, as illustrated in FIGs. 2A and 2B. The 
norm ||.|| can be chosen according to a severity of regulation to be imposed on the difference 

5 value. With this regulation, it can be ensured that extraneous area added to a boundary of an 
object through error in its defining pathline will be forced to an acceptable tolerance, providing 
an arbitrary control over, and therefore means of minimization of, variation in the difference 
error. 

Thus, in various embodiments of the present invention, the object outlines obey a 
10 maximum error threshold relative to an initial frame of definition. Various methods for 
converting two-dimensional images into three-dimensional images according to the present 
invention include selecting a severity of regulation for the distance spacing relationship to 
force errors in the object outlines to an acceptable tolerance. 



1 5 Tracking Virtual Objects: 

For purposes of the present description, the term "Virtual Objects" refers to objects 
that are defined in their entirety ignoring objects that may be in front of them. By way of 
example, and referring to FIG. 3 A, an image 300 includes background and foreground 
objects. The background objects include a dome 301, a left wall 302, and a street clock 303. 

20 The foreground objects include a person 305 and a car 306 which are both partially out of 
view, or off screen. Referring to FIG. 3B, the dome 301, left wall 302, and street clock 303 
are defined as Virtual Objects and are shown with dashed outlines. Because the dome 301, 
left wall 302, and street clock 303 are defined as Virtual Objects, the portions of these 
objects that are revealed over time, as the foreground objects person 305 and car 306 move 

25 out of the way (FIG. 4B), will have the correct depth information or shape associated with 
them. Thus, defining objects as Virtual Objects takes into account the fact that objects may 
be momentarily obstructed or out of view. 

Referring to FIG. 4A, in order to provide correct shape to these objects, they can be 
drawn in their entirety even outside of the frame. In this example, the image 400 includes a 
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person 401 and a car 402 as foreground objects. The person 401 and the car 402 are defined 
as virtual objects with portions 403 and 404 of their respective outlines extending outside of 
the image area. Although the object portions that are outside the image frame may be, for 
the most part, unknown, basic estimations or approximations of their shape can be employed 

5 making it significantly easier for an operator creating a 3D motion picture from a 2D motion 
picture to then apply appropriate depth contours or other information to provide correct 
shapes to the entirety of the objects. As a consequence, shapes are more easily maintained 
once the objects fully enter the image frame. By way of example, FIG. 4B shows a 
subsequent frame when both the person 401 and the car 402 have moved entirely into the 

1 0 frame with their respective shapes remaining as they were previously defined. 

According to various embodiments of the present invention, Virtual Objects are 
tracked automatically along with the image objects as objects move frame by frame over 
time, for example, by employing an image data tracking technique. For frames where some 
pathline (outline) segments of image objects are outside the image frame area, the Virtual 

15 Object outline is still a complete outline. As such an object moves into the image frame, the 
object can remain "connected together" as a Virtual Object and move in its entirety. Thus, in 
various embodiments of the present invention, a virtual object corresponding to the object is 
defined such that the virtual object is tracked and the object outlines are generated independent 
of whether the object is obstructed from view in a subsequent frame by another object moving 

20 into the foreground. 

Tracking Sub-Objects: 

Objects can be partitioned as "Sub-Objects" for the purpose of applying multiple 
areas of depth information to individual objects. Sub-objects can be of any shape or size 

25 and, in some embodiments, are contained completely within the associated full object. 
Visually, sub-objects are pieces of the surface of the full object that can be, but are not 
necessarily, shaped differently than the general surface of the full object. The depth values 
associated with a full object and a sub-object associated with the full object match along the 
boundary (or portion of the boundary) of the sub-object contained within the full object so 

30 that there is a continuous variation in depth across the boundary from the surface defined by 
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a sub-object to the general surface of the full object. One example of this could be the depth 
details within a person's face. Each feature within a person's face has certain depth 
characteristics, such as a person's eyes being somewhat recessed, the nose protruding, etc. 
Thus, sub-objects defined within objects facilitate the application of separate algorithms to 
5 the sub-object sections for greater flexibility in recreating the depth of otherwise flat 2D 
surfaces. 

According to various embodiments of the present invention, Sub-Objects are tracked 
automatically along with the image objects as objects move frame by frame over time, for 
example, by employing an image data tracking technique. This allows Sub-Objects to 

10 automatically re-conform in the same manner as Virtual Objects. By way of example, and 
referring to FIG. 5A, an image 500 shows a car 501 (with a dashed outline) entering into the 
frame. In this example, a back lid 502 of the car 501 is defined as a sub-object. Separate 
depth shaping can be applied to the sub-object even though a portion of the object has not 
fully entered the frame. Referring to FIG. 5B, as the car 501 enters the image frame, the 

15 sub-object is tracked and its contour depth shape information adjusted as necessary frame by 
frame. Thus, in various embodiments of the present invention, one or more sub-objects 
contained within an object are defined such that the sub-objects are linked together and tracked 
as the object moves or changes from frame to frame. 

Sub-objects can also be defined as Virtual Objects. Referring again to FIG. 5B, the 

20 back lid 502 can be defined as a Virtual Object the outline of which is shown by solid 
segment 503 which represents a visible portion of the object and by dashed segment 504 
which represents a portion of the object hidden by the bumper of the car 501 . 

Tracking Contour Points and Pathlines: 

25 In the 2D to 3D conversion process, contour information is used to provide shape to 

objects within images. By way of example, U.S. Patent No. 6,515,659 to Kaye et al. entitled 
"Method And System For Creating Realistic Smooth Three-Dimensional Depth Contours 
From Two-Dimensional Images" describes techniques for applying variable but smooth depth 
contouring to image objects. Since objects defined in an image can be of any shape or size, the 

30 ability to apply contour information in such a manner results in a higher degree of realism in 
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reconstructing graphical objects in three dimensions. By way of example, points and/or paths 
applied to objects provide convenient tools for creating crests, troughs or the like during the 
process of depth contouring the objects. This allows arbitrary shaping in the reconstruction of 
a three-dimensional surface of an object and object shaping with much greater depth detail, 

5 thereby producing an object with a surface that appears curved and shaped in three dimensions 
and imparting a high level of realism to the appearance of the final product. 

A variety of control functions, interactive interfaces, etc. can be utilized for 
producing a three-dimensional shape that will make a particular object appear as a realistic 
three-dimensional recreation. By way of example, and referring to FIGs. 6A-6F, points 

10 and/or paths can be applied to objects for creating crests, troughs or the like during the 
process of depth contouring the objects. For example, an operator can freehand draw (or 
otherwise apply) a point or pathline anywhere within the object to define a crest or ridge of a 
bulge function that is to be applied to the object. Bulges generated by the contour pathlines 
can be convex or concave in screen depth coordinates depending on the amplitude and 

1 5 positive or negative direction (polarity) of the algorithm that the operator applies to each of 
the independent bulge points and/or contour pathlines. By way of example, if the operator 
applies a single point or path, a single bulge will occur. If the operator applies multiple 
points or paths, multiple bulges will occur. 

In FIG. 6A, a boundary perimeter 650 for a randomly shaped object is shown with a 

20 point 652 positioned therein. The point 652 can be user or otherwise-defined and positioned 
anywhere within the object. Depending upon the polarity of the algorithm applied to the 
object, an outward or inward appearing bulge is applied to the object as shown in FIGs. 6B 
and 6C, respectively. 

As discussed above, multiple points can be applied to an object for creating crests, 
25 troughs or the like during the process of depth contouring the object. In FIG. 6D, for 
example, points 654 and 656 have been applied to the object, resulting in multiple contoured 
bulges being applied to the surface of the object. 

Also, as discussed above, one or more pathlines can be applied to an object for 
creating crests, troughs or the like during the process of depth contouring the object. In FIG. 
30 6E, for example, pathline 658 defines a ridge of a depth contour applied to the object. In 
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FIG. 6F, the pathline 658 f defines a different ridge applied to the object and, therefore, 
different depth contouring. The pathlines 658 and 658' can, but do not necessarily define an 
upper or lower most point of a depth ridge to be created. 

It should be appreciated that any number of points and/or pathlines can be applied to 
5 an object during the process of depth contouring the object. The amount of bulge, or 
amplitude, can then be adjusted accordingly to produce the desired and most realistic 
recreation of depth. 

As objects move and/or change shape in each frame over time, depth contour 
information must follow along with the object to maintain a correct and consistent shape. If 

10 the shape is not consistent across frames, the object shape may be erratic and noticeably 
distracting as the object moves. 

According to various embodiments of the present invention, depth contour 
information (such as Contour Points and Pathlines) are tracked automatically along with the 
image objects as objects move frame by frame over time, for example, by employing an 

1 5 image data tracking technique. This allows depth contour information to automatically re- 
conform in the same manner as Virtual Objects and Sub-Objects; and the shapes of objects 
automatically reform as the objects move and/or change from frame to frame. As an object 
moves from frame to frame, the depth contour information, such as contour pathlines, follow 
the movement of the object and reform their shape along with the changes in the object 

20 itself. Thus, in various embodiments of the present invention, one or more points contained 
within an object are associated with depth information and defined such that the one or more 
points track the object as the object moves or changes from frame to frame. In various 
embodiments of the present invention, one or more contour pathlines contained within an 
object are associated with depth contour information and defined such that the one or more 

25 contour pathlines track the object as the object moves or changes from frame to frame. 

Tracking Strategies: 

According to various embodiments of the present invention, the object outlines can be 
created employing a pixel image data tracking technique, a key frame curve interpolation 
30 technique, or a combination of pixel image data tracking and key frame curve interpolation 



-11 - 



Docket No. 016465-022 



techniques. With minimal user input, a consistent and accurate sequence of object pathlines 
across a sequence of frames can be generated by methods utilizing pixel image data tracking 
and/or key frame curve interpolation. It should be appreciated, however, that other methods 
for object segmentation in a sequence of images based on the object segmentation of an initial 
5 image or a proper subset of images in the sequence can be employed. 

With respect to maintaining and updating the shaping constructs of an object in a way 
that is consistent with the associated object, consistent topology and consistent geometry are 
the two most important considerations. Topologically, constructs initially contained by the 
closure of the object pathline should remain inside of this planar set over time. FIGs. 7A and 
10 7B show examples of inconsistent topology and consistent (or preserved) topology, 
respectively. Geometrically, object surface shaping should remain consistent with the 
geometry of the user's initial or key definitions. FIGs. 8 A and 8B show examples of 
inconsistent surface shaping and consistent surface shaping, respectively. 

1 5 Key Frame Curve Interpolation Strategy: 

Using an appropriate family of curve approximation (b-spline approximations, bezier 
approximations, etc.), a curve shape can be defined in several user-selected "key frames" of an 
image sequence and likely shapes for the undefined frames can be interpolated from the "key 
frame" data. Although implementation of these interpolative methods generally generate 

20 curves in the undefined frames that both closely approximate the actual outline of the image 
object and are consistent with the "key frame" data, the various constructs of the object can be 
tracked appropriately, as discussed above. 

According to various embodiments of the present invention, interpolative techniques 
can be employed to consider the input "key frame" data and to generate a discrete and 

25 sequential mapping from one "key frame" to the next. This map is then applied to each time 
step in between two "key frames" to approximate the missing data in the frames between. This 
map is, by construction, a homeomorphism. Referring to FIG. 9, this means that the topology 
of point-sets that the map is applied to is preserved; thus, given the interpolative mapping, M, 
and a set A and point p such that: 

30 M:R n ^>R\ AcR*, p e A. 
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results in 

p € A => M{p) e M(A). 

This means that if the interpolative map constructed from the object pathline "key 
frame" data is applied to the curve/point-based constructs contained in the object, these 
5 constructs will remain inside of the object (as in FIG. 7B). 

The interpolative, by construction, also preserves the relative geometry of a set, so that 
if this map is applied to internal curve/point-based constructs, these constructs will follow, in 
an appropriate geometry, the change in character of the object pathline (as in FIG. 8B). This 
preserves the spirit of the user's initial object shape definition, while requiring the user to 
10 define the shaping constructs only in the initial frame - this data is not needed in the "key 
frames", only the pathline data for the object is needed in the "key frames". 

It is important to note that this strategy does not use any image data, and therefore, can 
be applied to Virtual Objects with the same degree of accuracy as fully visible objects. 

1 5 Image Data Tracking Strategy: 

According to various embodiments of the present invention, an object pathline in an 
initial frame is correlated with the actual image data of the object and the image data is used to 
track the transition of individual image object pixel points across a sequence of frames while 
preserving regularity in the topology and the geometry of constructs within the object (such as 
20 sub-objects and shaping contours). Some example image data tracking strategies for regulating 
these two properties are set forth below. 

One method is to track an object pathline from one image to the next using image data. 
A general homeomorphic mapping can then be constructed from the initial pathline to the 
updated pathline and this map can be applied to all constructs contained in the object, 
25 producing results similar to those discussed in the previous section. 

Another method is to use image data to track all object constructs separately and then 
clamp any constructs that violate the geometry and topology constraints to the boundary of the 
object, i.e., the object pathline. 

Image data tracking techniques generally ensure some level of uniqueness, i.e., if the 
30 same pixel is passed in to the algorithm multiple times, the same solution will be given each 

-13- 



Docket No. 016465-022 



time. This means that boundary conditions are preserved: for example, if a sub-object has part 
of its pathline in common with the object, that section of the sub-object pathline will be moved 
exactly as the associated object pathline section. However, interior containment may not be 
preserved so that a contour point inside the area defined by the object pathline my not be inside 
5 the updated area after tracking, especially if the tracking algorithm updates the object outline in 
the next image but fails to find the contour point in that image. Similar issues may occur with 
sub-objects, producing results similar to those shown in FIG. 7 A. 

According to various embodiments of the present invention, and referring to FIG. 1 0, 
this problem can be addressed by clamping all data to the boundary of the object, intersecting 

10 the resultant construct with the resultant object pathline. In this way, the image data that is 
capable of being tracked can be used, preserving the general shaping of the object while 
updating the object and its constructs with minimal input from the user and efficiently 
generating visually correct and consistent objects. 

For some image objects, image data may be missing (e.g., for "hidden areas" of image 

15 objects). In such cases, information about the movement of those parts of the object pathline 
that can be tracked can be employed to approximate the movement of pathline sections lying 
over missing image data. To this end, Virtual Objects can be employed. Additionally, a well 
defined mapping can be defined by the movement of track-able portions of the object pathline 
and this mapping can then be applied to untrack-able portions of the pathline, allowing all 

20 portions of the object pathline to be tracked with an acceptable level of regularity. See, 
"Motion Edge Artifacts and Correction" section, supra, regarding object regulation. 

FIG. 11 shows an exemplary system 1100 suitable for implementing the various 
methods of the present invention. During the process of converting images into 3D images, 

25 objects can be defined in their entirety as Virtual Objects regardless of whether parts of their 
surfaces are hidden or not. These virtual objects may contain one or more sub-objects, as well 
as contour information in the form of points or pathlines for providing depth shape to the 
objects. Various principles of the present invention embody tracking techniques to track and 
maintain consistent object to object pathline spacing relationships, indicated at block 1103, 

30 which in turn, prevent (or substantially prevent) noticeable artifacts from occurring. An image 



-14- 



Docket No. 016465-022 



frame 1 101 (Frame A) is the first of a sequence of frames (Frame A, B, C, etc.). The first 3 
frames of the sequence are shown overlaid in one frame 1102, representing the consistent 
object to object pathline tracking according to the present invention. 

Various principles of the present invention are embodied in an interactive user interface 
5 and a plurality of image processing tools that allow a user to rapidly convert a large number of 
images or frames to create authentic and realistic appearing three-dimensional images. In the 
illustrated example system 1 100, the 2D-to-3D conversion processing, indicated at block 1 104, 
is implemented and controlled by a user working at a conversion workstation 1 105. It is here, 
at a conversion workstation 1 105, that the user gains access to the interactive user interface and 

10 the image processing tools and controls and monitors the results of the 2D-to-3D conversion 
processing. It should be understood that the functions implemented during the 2D-to-3D 
processing can be performed by one or more processor/controller. Moreover, these functions 
can be implemented employing a combination of software, hardware and/or firmware taking 
into consideration the particular requirements, desired performance levels, etc. for a given 

1 5 system or application. 

The three-dimensional converted product and its associated working files can be stored 
(storage and data compression 1 1 06) on hard disk, in memory, on tape, or on any other means 
of data storage. In the interest of conserving space on the above-mentioned storage devices, it 
is standard practice to data compress the information; otherwise files sizes can become 

20 extraordinarily large especially when full-length motion pictures are involved. Data 
compression also becomes necessary when the information needs to pass through a system 
with limited bandwidth, such as a broadcast transmission channel, for instance, although 
compression is not absolutely necessary to the process if bandwidth limitations are not an 
issue. 

25 The three-dimensional converted content data can be stored in many forms. The data 

can be stored on a hard disk 1107 (for hard disk playback 1124), in removable or non- 
removable memory 1 108 (for use by a memory player 1 125), or on removable disks 1 109 (for 
use by a removable disk player 1 126), which may include but are not limited to digital versatile 
disks (dvd's). The three-dimensional converted product can also be compressed into the 

30 bandwidth necessary to be transmitted by a data broadcast receiver 1110 across the Internet 
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1111, and then received by a data broadcast receiver 1112 and decompressed (data 
decompression 1113), making it available for use via various 3D capable display devices 1114 
(e.g., a monitor display 1118, possibly incorporating a cathode ray tube (CRT), a display panel 
1119 such as a plasma display panel (PDP) or liquid crystal display (LCD), a front or rear 
5 projector 1 120 in the home, industry, or in the cinema, or a virtual reality (VR) type of headset 
1121.) 

Similar to broadcasting over the Internet, the product created by the present invention 
can be transmitted by way of electromagnetic or radio frequency (RF) transmission by a 
radio frequency transmitter 1115. This includes direct conventional television transmission, 

10 as well as satellite transmission employing an antenna dish 1116. The content created by 
way of the present invention can be transmitted by satellite and received by an antenna dish 
1117, decompressed, and viewed on a monitor display 1118, possibly incorporating a 
cathode ray tube (CRT), a display panel 1119 such as a plasma display panel (PDP) or liquid 
crystal display (LCD), a front or rear projector 1 120 in the home, industry, or in the cinema, 

15 or a virtual reality (VR) type of headset 1121. If the three-dimensional content is broadcast 
by way of RF transmission, the receiver 1122, can in feed decompression circuitry directly, 
or feed a display device directly. Either is possible. It should be noted however that the 
content product produced by the present invention is not limited to compressed data formats. 
The product may also be used in an uncompressed form. Another use for the product and 

20 content produced by the present invention is cable television 1 123. 

Although the present invention has been described in terms of the example 
embodiments above, numerous modifications and/or additions to the above-described 
embodiments would be readily apparent to one skilled in the art. It is intended that the scope 
of the present invention extends to all such modifications and/or additions. 
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